Interpreting Image Classification of Keras InceptionV3¶
This notebook examines the predictions of an image classfifier (DNN InceptionV3) using two interpretation methods:
- LimeImageExplainer (LIME)
- Class Activation Maps (CAM)
- First, LIME and CAM will be applied on a single prediction to make their concepts easier to understand.
- In the following section, both methods are systematically applied to a group of predictions and compared using their masks
- The Images used in this notebook can be downloaded from ImageNet: Download from here
In [1]:
#process not on GPU
import os
os.environ['CUDA_VISIBLE_DEVICES'] = ''
Loading packages¶
In [2]:
# Keras
import keras
from keras.applications.inception_v3 import InceptionV3, preprocess_input, decode_predictions
from keras.preprocessing.image import load_img, img_to_array
# Tools requiered for visualizing and processing
import matplotlib.pyplot as plt
import pandas as pd
In [3]:
model_InceptionV3 = InceptionV3(weights = 'imagenet')
Getting started with a single prediction of the model¶
In [4]:
# Load image and scale to 299x299
image_raw = load_img("/home/common_files/datasets/imagenet/pictures/n01484850/n01484850_10798.jpg",target_size = (299,299))
image_raw
Out[4]:
In [5]:
# Preprocess image in an array to make it compatible with the Keras model and reshape it.
image = img_to_array(image_raw)
image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))
image = preprocess_input(image)
In [6]:
# Predict label of image
prediction = model_InceptionV3.predict(image)
pd.DataFrame(decode_predictions(prediction, top=3)[0],columns=['col1','class','probability']).iloc[:,1:]
Out[6]:
Loading LIME¶
In [7]:
import lime
from lime.lime_image import LimeImageExplainer
# Requiered segmentation package for creating superpixel of image
from skimage.segmentation import mark_boundaries, quickshift
import numpy as np
Create explaination¶
In [8]:
explainer = LimeImageExplainer()
In [9]:
# This part is optional and is listed to make clear the idea behind LimeImageExplainer. This means Lime does this automatically in the back.
# Lime is creating superpixels using the segmentation algorithm quickshift like below, and predicts the image user-defined-times by turning superpixel on and off
segments = quickshift(image_raw, kernel_size=4, max_dist=200, ratio=0.2,random_seed=42)
superpixel = mark_boundaries(image_raw, segments)
plt.imshow(superpixel)
print('Number of superpixel: ',len(np.unique(segments)))
In [10]:
# pass image as (299,299,3)-Array 299x299Pixel in RGB
# top_labels = 3 -> create explanation for top three labels predicted by the model for image
# num_samples = 500 -> create 500 similiar pictures by turning superpixel on and off
# random_seed = 42 -> for reproducability
explanation = explainer.explain_instance(image[0], model_InceptionV3.predict, top_labels=3, num_samples=500, random_seed=42)
In [11]:
# Show top five (by default) Superpixel with positive weights supported decision making
temp, mask = explanation.get_image_and_mask(explanation.top_labels[0], positive_only=True, hide_rest=True)
top_superpixels = mark_boundaries(temp /2 + .5,mask)
plt.imshow(top_superpixels)
Out[11]:
In [12]:
# explanation.top_labels[0] -> index of highest probable predicted class
temp, mask = explanation.get_image_and_mask(explanation.top_labels[0], positive_only=False, hide_rest=False)
top_superpixels = mark_boundaries(temp /2 + .5,mask)
plt.imshow(top_superpixels)
Out[12]:
In [13]:
temp, mask = explanation.get_image_and_mask(explanation.top_labels[0], positive_only=False, num_features=len(np.unique(segments)),hide_rest=False)
top_superpixels = mark_boundaries(temp /2 + .5,mask)
plt.imshow(top_superpixels)
Out[13]:
Create explanations for a group of predictions¶
- We predicted 810 992 images (from the ImageNet database) without retraining the model
- Result: Accuracy of 86.9 %
- At the end we created a dataset containing all false predicted images
In [37]:
results_images_df = pd.read_hdf('results-inceptionv3-1.h5',key='pictures')
results_images_df.sort_values(['confidence_difference'], ascending=False)
Out[37]:
In [14]:
%%html
<style>
table {float:left}
</style>
Columns of dataset explained¶
| Column | Explanation |
|---|---|
| label_id | index of actual label |
| label_name1 | encoded actual label |
| label_name2 | decoded actual label |
| prediction_id | index of predicted label |
| prediction_name1 | encoded predicted label |
| prediction_name2 | decoded predicted label |
| file_name | directory of image |
| confidence_in_prediction | probability for predicted label by model |
| prediction_in_top_5 | True, if actual label is in top five of predicted classes, otherwise False |
| confidence_in_label | probability for actual label predicted by model |
confidence_difference |confidence_in_prediction - confidence_in_label|
Define methods for codes above¶
In [15]:
# To load images
def get_image(image_dir):
image = load_img("/home/common_files/datasets/imagenet/pictures/"+image_dir,target_size = (299,299))
return image
In [16]:
# Preprocess image in an array to make it compatible with the Keras model and reshape it.
def preprocess_image(image_raw):
image = img_to_array(image_raw)
image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))
image = preprocess_input(image)
return image[0]
In [17]:
# Create explanation of Image
def explanation_for_instance(image,model):
explanation = explainer.explain_instance(image,model.predict,top_labels = 1,num_samples=500,random_seed=42)
return explanation
In [18]:
# Mask explanation and return top ten superpixel for most probable prediction of model and hide rest of image
def get_explained_image(explanation, positive_only, hide_rest):
temp, mask = explanation.get_image_and_mask(explanation.top_labels[0], positive_only = positive_only, num_features=10, hide_rest=hide_rest)
bounderies = mark_boundaries(temp /2 + .5,mask)
return bounderies
Create explanations in loop¶
- We want to create explanations for only images where confidence_difference is higher than 0.98 since we have more than 100 000 false predictions
- And it takes to much time generate an explanation for an image
In [34]:
# Order dataframe by confidence_difference descending to make iteration more efficent
results_images_df = results_images_df.sort_values(['confidence_difference'], ascending=False)
# Create explanation in loop
for i in range(len(results_images_df)):
file_name = results_images_df.iloc[i]['file_name']
image_raw = get_image(file_name)
image = preprocess_image(image_raw)
# Create explanation for predicted label
explanation = explanation_for_instance(image, model_InceptionV3)
# get explained image with 10 positive weighted superpixel and hide rest
explained_image_positive_and_hidden = get_explained_image(explanation, positive_only = True, hide_rest= True)
# get explained image with 10 positive or negative weigthed superpixel, hide_rest = False
explained_image_pos_neg = get_explained_image(explanation, positive_only = False, hide_rest= False)
#plt.savefig('../Output_Images/Predicted_ {}, Actual_ {}'.format(results_images_df.iloc[i]['prediction_name2'],results_images_df.iloc[i]['label_name2']))
#plot only first 10
if i < 10:
fig, ax = plt.subplots(1,3, figsize = (15,15))
ax[0].imshow(image_raw)
ax[0].set_title('Predicted: {}, Actual: {}'.format(results_images_df.iloc[i]['prediction_name2'],results_images_df.iloc[i]['label_name2']), fontsize =12)
ax[1].imshow(explained_image_positive_and_hidden)
ax[1].set_title('10 most positive superpixel, rest hidden')
ax[2].imshow(explained_image_pos_neg)
ax[2].set_title('Positive and negative weighted superpixels')
plt.show()
if results_images_df.iloc[i]['confidence_difference'] < 0.98:
break
if i >=10:
break
LIME vs. Class Activation Maps¶
In this section Class Activation Maps are generated and compared with the results of the LimeImageExplainer.
Class Activation Maps are simple to implement and can help to get the discriminative image regions used by a CNN to identify a specific class in the image.
Class Activation Maps are often called as Attention Maps
Get Class Activation Map for a single prediction¶
In [19]:
# import keras-vis
import vis
from vis.utils import utils
from vis.visualization import visualize_saliency, overlay, visualize_cam,visualize_cam_with_losses
from keras import activations
import os
In [20]:
image_raw = load_img("/home/common_files/datasets/imagenet/pictures/n01910747/n01910747_15064.jpg",target_size = (299,299))
plt.imshow(image_raw)
# n01910747_13855.jpg
#n01484850/n01484850_10798.jpg
# n01560419/n01560419_1316.jpg
Out[20]:
In [20]:
def cam(img_path):
import keras
import numpy as np
import pandas as pd
from keras.applications.inception_v3 import InceptionV3
import matplotlib.image as mpimg
from keras import backend as K
import matplotlib.pyplot as plt
%matplotlib inline
K.clear_session()
print('here_1')
model = InceptionV3()
img=mpimg.imread(img_path)
plt.imshow(img)
from keras.preprocessing import image
img = image.load_img(img_path, target_size=(299, 299))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
print('here_2')
from keras.applications.inception_v3 import preprocess_input
x = preprocess_input(x)
preds = model.predict(x)
predictions = pd.DataFrame(decode_predictions(preds, top=3)[0],columns=['col1','category','probability']).iloc[:,1:]
print('decoded predictions',predictions)
argmax = np.argmax(preds[0])
print('argmax: ', argmax)
output = model.output[:, argmax]
last_conv_layer = model.get_layer('conv2d_94')
grads = K.gradients(output, last_conv_layer.output)[0]
pooled_grads = K.mean(grads, axis=(0, 1, 2))
iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])
pooled_grads_value, conv_layer_output_value = iterate([x])
print('here_3')
for i in range(192):
conv_layer_output_value[:, :, i] *= pooled_grads_value[i]
heatmap = np.mean(conv_layer_output_value, axis=-1)
heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)
import cv2
img = cv2.imread(img_path)
heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))
heatmap = np.uint8(255 * heatmap)
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)
hif = .8
superimposed_img = heatmap * hif + img
output = 'test.jpeg'
cv2.imwrite(output, superimposed_img)
img=mpimg.imread(output)
plt.imshow(img)
plt.axis('off')
plt.title(predictions.loc[0,'category'].upper())
return None
In [30]:
cam('/home/common_files/datasets/imagenet/pictures/n01910747/n01910747_15064.jpg')
In [21]:
#Preprocess input
image = img_to_array(image_raw)
image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))
image = preprocess_input(image)
In [22]:
# predict
prediction = model_InceptionV3.predict(image)
print(pd.DataFrame(decode_predictions(prediction, top=3)[0],columns=['col1','class','probability']).iloc[:,1:])
# get corrosponding imagenet index for prediction
argmax = np.argmax(prediction[0])
print('\nImageNet Index for prediction: ',argmax)
In [23]:
# get last layer
prediction_layer = model_InceptionV3.get_layer('predictions')
print(prediction_layer)
In [24]:
# get index of last layer
#from keras import activations
prediction_layer_idx = utils.find_layer_idx(model_InceptionV3, 'predictions')
print(prediction_layer_idx)
#model_InceptionV3.layers[prediction_layer_idx].activation = activations.linear
#model_InceptionV3 = utils.apply_modifications(model_InceptionV3)
In [25]:
# filter_indices is the corresponding label index for great_white_share
cam = visualize_cam(model=model_InceptionV3, layer_idx=prediction_layer_idx, filter_indices=argmax, seed_input=image)#, backprop_modifier='relu', grad_modifier='absolute')
plt.imshow(cam)
cam.shape
Out[25]:
In [26]:
plt.imshow(overlay(np.array(image_raw),cam, alpha=0.4))
Out[26]:
In [27]:
# Get the red area of the attention_map (most positive area for classification of cnn)
# initialize 3D-array of zeros 299x299Pixel RGB
cam_mask = np.zeros((299,299))
# fill array, where blue channel is 0
mask = cam[:,:,2] == 0
# Set to 1 the pixels where mask is True
cam_mask[mask] = 1
plt.imshow(cam_mask)
Out[27]:
In [28]:
# get lime mask for top 5 superpixel of actual label
#first create explanation the image index of great white shark (labels = (2,))
explanation = explainer.explain_instance(image[0], model_InceptionV3.predict,num_samples = 500, labels = (argmax,), top_labels = None)
temp, lime_mask = explanation.get_image_and_mask(argmax, positive_only = True, hide_rest = True)
boundaries = mark_boundaries(temp / 2 + .5, lime_mask)
plt.imshow(lime_mask)
Out[28]:
Compare both masks¶
In [29]:
# Compare both masks
masks = np.zeros((299,299,3))
masks[:,:,0] = cam_mask
masks[:,:,1] = lime_mask
plt.imshow(masks)
Out[29]:
In [30]:
# Pixel in intersection (Yellow)
pixel_intersection = ((lime_mask + cam_mask) == 2).sum()
# Totalpixel_attentionMap without intersection
pixel_cam = (cam_mask == 1).sum() - pixel_intersection
# Totalpixel Lime without intersection
pixel_lime = (lime_mask == 1 ).sum() - pixel_intersection
# Get match in percentage (Yellow)
match_in_percent = pixel_intersection / (pixel_lime + pixel_cam + pixel_intersection)
fig, ax = plt.subplots(1,4, figsize = (15,15))
ax[0].imshow(image_raw)
ax[0].set_title('Original Image')
ax[1].imshow(boundaries)
ax[1].set_title('Lime explanation')
ax[2].imshow(overlay(np.array(image_raw),cam, alpha=0.5))
ax[2].set_title('Class Activation Map')
ax[3].imshow(masks)
ax[3].set_title('Overlay of LIME & CAM')
plt.show()
print('Overlay of LIME and Class Activation Map:\n')
print('Pixel in intersection (Yellow): {} \nPixel only in CAM (Red): \t{} \nPixel only in LIME (Green): \t{} \nMatch in percentage (Yellow): \t{}'.format(pixel_intersection,pixel_cam, pixel_lime,match_in_percent))
Get code above in methods and create explanations for group of predictions¶
In [39]:
plt.imshow(explanation.segments)
Out[39]:
In [32]:
def get_mask_lime(path,image_index):
#Preprocess input
image_raw = load_img(path,target_size = (299,299))
image = img_to_array(image_raw)
image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))
image = preprocess_input(image)
explanation = explainer.explain_instance(image[0], model_InceptionV3.predict,num_samples = 500, labels = (image_index,), top_labels = None)
temp, lime_mask = explanation.get_image_and_mask(image_index, positive_only = True, hide_rest = True)
boundaries = mark_boundaries(temp / 2 + .5, lime_mask)
#plt.imshow(lime_mask)
return lime_mask, boundaries
In [33]:
def get_mask_cam(path,image_index):
image_raw = load_img(path,target_size = (299,299))
# Preprocess image in an array to make it compatible with the Keras model and reshape it.
image = img_to_array(image_raw)
image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))
image = preprocess_input(image)
cam_mask = np.zeros((299,299))
cam = visualize_cam(model_InceptionV3, layer_idx=prediction_layer_idx, filter_indices=image_index, seed_input=image)
# fill array, where blue channel is 0
mask = cam[:,:,2] == 0
# Set to 1 the pixels where mask is True
cam_mask[mask] = 1
return cam_mask, cam
In [34]:
def compare_masks(lime_mask,cam_mask):
# Pixel in intersection (Yellow)
# Compare both masks
masks = np.zeros((299,299,3))
masks[:,:,0] = cam_mask
masks[:,:,1] = lime_mask
pixel_intersection = ((lime_mask + cam_mask) == 2).sum()
# Totalpixel_attentionMap without intersection
pixel_cam = (cam_mask == 1).sum() - pixel_intersection
# Totalpixel Lime without intersection
pixel_lime = (lime_mask == 1 ).sum() - pixel_intersection
# Get match in percentage (Yellow)
match_in_percent = pixel_intersection / (pixel_lime + pixel_cam + pixel_intersection)
return masks,pixel_intersection,pixel_lime,pixel_cam, match_in_percent
In [35]:
import json
classJSON = "/home/viscif/.keras/models/imagenet_class_index.json"
with open(classJSON) as json_file:
label_list = json.load(json_file)
invers_label_list = {}
for class_ID, value in label_list.items():
invers_label_list[value[0]] = int(class_ID)
In [36]:
def get_prediction(path):
image_raw = load_img(path, target_size = (299,299))
image = img_to_array(image_raw)
image = image.reshape((1, image.shape[0], image.shape[1], image.shape[2]))
image = preprocess_input(image)
prediction = model_InceptionV3.predict(image)
predicted_label = decode_predictions(prediction)[0][0][0]
predicted_label_name = decode_predictions(prediction)[0][0][1]
return prediction[0], predicted_label, predicted_label_name
In [37]:
import os
base_path = '/home/common_files/datasets/imagenet/pictures'
resultsDF = pd.DataFrame()
i = 0
for dir_, subfolder, file_name in os.walk(base_path):
for class_name in subfolder:
class_paths = os.path.join(dir_,class_name)
for dir_, s, file_name in os.walk(class_paths):
for file in file_name:
i += 1
#print('file: ', file)
path_to_image = os.path.join(dir_, file)
image_raw = load_img(path_to_image, target_size=(299,299))
image_index = invers_label_list[class_name]
actual_label_name = label_list[str(image_index)][1]
lime_mask, boundaries = get_mask_lime(path_to_image, image_index)
cam_mask, cam_map = get_mask_cam(path_to_image, image_index)
both_masks, pixel_intersection, pixel_lime, pixel_cam, match_in_percent = compare_masks(lime_mask, cam_mask)
prediction,predicted_label,predicted_label_name = get_prediction(path_to_image)
data = {
'lime_mask' : [lime_mask],
'cam_mask' : [cam_mask],
'file_name': [file],
'image_class_ID' : [image_index],
'percentage_lime_in_cam' : [match_in_percent*100],
'pixel_intersection_lime_cam' : [pixel_intersection],
'pixel_only_lime' : [pixel_lime],
'pixel_only_lime' : [pixel_cam],
'actual_label' : [class_name],
'actual_label_name' : [actual_label_name],
'predicted_label': [predicted_label],
'predicted_label_name' : [predicted_label_name],
'prediction' : [prediction]
}
resultsDF = resultsDF.append(pd.DataFrame(data),ignore_index = False)
#plot only first 10
if i <= 10:
fig, ax = plt.subplots(1,4, figsize = (15,15))
ax[0].imshow(image_raw)
ax[0].set_title('Original Image')
ax[1].imshow(boundaries)
ax[1].set_title('Lime explanation for actual label')
ax[2].imshow(overlay(np.array(image_raw),cam_map, alpha=0.4))
ax[2].set_title('Class Activation Map')
ax[3].imshow(both_masks)
ax[3].set_title('Overlay of LIME & CAM')
plt.show()
print('Overlay of LIME and Class Activation Map:\n')
print('Predicted label: \t\t{} \nActual label: \t\t\t{} \nPixel in intersection (Yellow): {} \nPixel only in CAM (Red): \t{} \nPixel only in LIME (Green): \t{} \nMatch in percentage (Yellow): \t{}'.
format(predicted_label_name,actual_label_name,pixel_intersection,pixel_cam, pixel_lime,match_in_percent))
if i == 11:
break
resultsDF.to_hdf('results_lime_cam.h5', key = class_name)
In [ ]: